In recent years, digital still camera (DSC), digital video (DV), and scanner have been widely utilized to take pictures and to record progress of colorful life as the technology of digital image acquisition advances rapidly. However, recording a digital image requires many processing steps which include white balance adjustment, gamma correction, data compression, and so forth. Interpolation and reconstruction of CFA, which is the so called “demosaicking”, is one of the most important processing step.
There are many kinds of patterns adopted in CFAs. Among them, Bayer CFA of three colors is the most common one, which filters red, green, and blue. Some other CFAs of four colors filter cyan, magenta, yellow, and green. Taking the commonly adopted Bayer CFA as an example, a typical CFA is described below.
Each pixel of a color image requires at least three basic colors to construct the hues of original color image. Red, green, and blue three basic colors are often adopted in a typical computer image. If the color image of a scenery is to be truly presented, at least three image sensors are required to record the three basic colors of each pixel. In order to reduce the form factor and to save the hardware cost, the majority of image acquisition systems only contain a single image sensor with a CFA. However, a CFA is a regular pattern of color filters that allow the image sensor to detect only one basic color at each pixel location. This means that an image acquisition system must reconstruct three full colors for each pixel, including the other two undetected colors. This is the so called “demosaicking” process.
Since the bayer CFA pattern was disclosed in 1976, many demosaicking methods have been proposed. In general, the demosaicking methods are roughly separated into two kinds. In the first kind, no detection of edges is performed. Instead, same mathematical formula is repeatedly applied to each pixel. In the second kind, prior knowledge of some geometric patterns is utilized to detect if edges exist between a target pixel and other pixels in the neighborhood. If an edge is found, an appropriate demosaicking algorithm will be selected and applied to reconstruct the missing colors of the target pixel depending on the feature of the edge.
There is no detection of edges for the first kind of demosaicking methods. The simplest way to generate a full color image is to adopt a traditional bilinear interpolation. It utilizes the values of a missing color detected by the neighboring pixels to reconstruct the value of the missing color for the target pixel through some form of linear interpolation. Though the above method is simple and easy to implement, the bilinear interpolation usually results in bad by-products accompanied with the demosaicking process, e.g., the appearance of colored artifacts and fringes.
In 2000, Pei proposed a demosaicking method (“Effective color interpolation in CCD color filter array using signal correlation,” IEEE Image Processing) based on the high correlation between the R, G, B channels. By defining Kr equal to the value difference between G and R and Kb equal to the value difference between G and B, a theory was then developed by treating these two values as continuous constants to derive mathematic formulas for the other two missing colors.
According to U.S. Pat. No. 4,642,678, Cok disclosed a simple phenomenon about spectrum relationship between different color planes. Within small neighborhoods in a image, the color ratios of red/green and blue/green are very similar. This phenomenon has been applied to many schemes to interpolate the missing color value through the information obtained from other colors planes. Besides the color ratios, color differences are also used by many other schemes that adopted a similar concept for interpolation of missing color values. However, the schemes described above still cannot resolve the unwanted colored fringes and hue (color ratio) shifts introduced in areas of image detail.
In 2002, Gunturk et al (“Color plane interpolation using alternating projections,” IEEE Trans. Image Processing) proposed an effective method for interpolation of missing color values. The authors used original CFAs to form constraint sets. The junction of these two constraint sets represents the space of acceptable solution and can be estimated through the above mentioned spectrum relationship and an alternating projection. The missing color values of each pixel are then derived from the junction of these two constraint sets. In 2003, another method for interpolating missing color values was proposed by Lu et al (“On new method and performance measures for color filter array,” IEEE Trans. Image Processing). In this method, the authors took into consideration of the color relationships of a target pixel with its neighboring pixels in four different directions, i.e., up, down, left and right. The bigger the difference of the brightness between target pixel and same-color pixel in the interpolation direction is, the smaller the spatial relationship between them. A mathematical formula for the missing color is then derived. Moreover, Lu also adopted an adaptive median filtering method for post processing of demosaicking.
According to US Patent Publication 2003/0215159, Okuno et al disclosed a pixel interpolation device which includes an interpolation pattern table. The interpolation pattern table outputs interpolation direction data designating interpolation directions. Each interpolated pixel data is calculated based on a pixel data located in an interpolation direction designated by an interpolation direction data.
For the second kind of demosaicking methods, a detection of edges is performed. In 1986, Cok (U.S. Pat. No. 4,630,307) proposed a demosaicking method based on pattern recognition (PR). Firstly, a plurality of different interpolation routines are provided to generate appropriate interpolated values, and to complete respective geometrical features. And, local neighborhoods of a target pixel around the interpolation location are examined to see if they match with some existing geometrical features, such as edge, line, and corner. If an existing geometrical feature is detected, then the interpolation routine appropriate for the detected feature is applied to generate an interpolated signal value. If there is no match, the simple bilinear interpolation method will be applied to estimate the missing color value. Generally speaking, using spatial relationship of neighboring pixels to interpolate the missing colors will improve the appearance of colored fringes. This kind of interpolation methods is better than those methods in which edges are ignored. For example, Laroche et al (U.S. Pat. No. 5,373,322) proposed an edge pattern classification method to select a preferred orientation for the interpolation of missing colors. The interpolation is then performed upon values selected along with the preferred orientation.
According to U.S. Pat. No. 5,629,734, Hamilton et al proposed a demosaicking apparatus in which Laplacian second-order values and gradient values are utilized as detection tools of edges to select a preferred orientation for the interpolation of missing color value.
Besides, some people adopt a combined scheme by using at least two demosaicking processes to obtain sharp color edges and to reduce undesirable by-products resulted from the demosaicking process. In 1999, Kimmel (“Demosaicking: image reconstruction from color CCD samples,” IEEE Trans. Image Processing) adopted an edge-oriented technique to get the missing colors. The authors used some edge-patterns recognitions to interpolate color values or color ratios of four neighboring pixels. At the same time, an inverse diffusion process is applied to suppress the unwanted by-products accompanied with the demosaicking process. In 2001, Li et al (“New edge-directed interpolation,” IEEE Trans. Image Processing) disclosed a method for detecting edges. Based on geometric duality and edge-orientated property of the covariance between low-resolution CFA image and high resolution post-demosaicking image as well as the brightness difference between different color planes, the missing two colors are interpolated.
There are plenty of existing demosaicking methods and apparatuses. They are mainly used for producing high-quality post-demosaicking images. However, a practical demosaicking method and apparatus must be able to improve image sharpness and colored fringes without unduly increasing hardware cost.